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Preface 


This  report  describes  a  pilot  evaluation  of  The  Tactile  Situation  Awareness  System  (TSAS) 
during  simulated  flight.  The  objective  was  to  evaluate  the  ability  of  TSAS  to  improve  a  pilot’s 
hover  stability  in  a  simulated  degraded  visual  environment  (DVE  or  “brownout”)  condition.  The 
Department  of  Defense  (DoD)  Coalition  Warfare  Program  (CWP)  funded  the  U.S.  Army 
Aeromedical  Research  Laboratory  (USAARL)  to  refine  the  Tactile  Situation  Awareness  System 
(TSAS)  and  deliver  it  to  aviation  rotary-wing  operators  (Lawson  and  Rupert,  2014).  The  refined 
system  was  then  tested  by  Chesapeake  Technology  International  (CTI)  and  the  test  results 
provided  to  the  Naval  Aviation  Center  for  Rotorcraft  Advancement  (NACRA)  at  the  conclusion 
of  CTI’s  SBIR  Project  Contract  No.:  N68335-09-C-0025.  The  findings  are  reiterated  in  this 
report  with  permission  of  NACRA  and  CTI. 


v 


This  page  intentionally  left  blank. 


vi 


Table  of  contents 


Page 

Introduction . 1 

TSAS  test  results . 1 

Objective . 1 

Methods . 1 

Findings . 6 

Conclusion . 11 

References . 112 

Acronyms . 13 

List  of  figures 

1 .  Representation  of  hover  course . 3 

2.  Handling  Qualities  Rating  Form . 4 

3.  Post- flight  questionnaire . 5 

4.  Average  hover  deviation  of  all  CVE  flights . 7 

5.  Average  hover  deviation  of  all  DVE  flights . 8 

6.  Average  HQR . 10 

List  of  tables 

1.  Pilot  ADS33  hover  deviation  statistics . 9 

2.  HQR  ratings . 10 

3.  Hover  deviation  during  DVE . 11 

vii 


This  page  intentionally  left  blank. 


Introduction 


The  idea  of  employing  tactile  displays  to  maintain  spatial  orientation  and  situation  awareness 
during  flight  was  introduced  at  a  1989  Advisory  Group  for  Aerospace  Research  &  Development 
(AGARD)  meeting  in  Copenhagen  (Rupert,  Mateczun,  and  Guedry,  1990).  Subsequent  flight 
testing  in  fixed-wing  (Rupert,  Guedry,  and  Reschke,  1994)  and  rotary-wing  aircraft  (Raj,  Suri, 
Braithwaite,  and  Rupert,  1998)  demonstrated  that  continuous,  intuitive  orientation  infonnation 
could  be  provided  via  tactile  cueing. 

The  Tactile  Situation  Awareness  System  (TSAS)  used  for  the  evaluations  described  in  this 
report  is  made  up  of  a  commercial  processor  (that  detennines  if  the  aircraft  is  in  a  potentially 
dangerous  flight  regime  and  passes  simple  intuitive  commands  to  a  pilot  via  a  lightweight  vest), 
a  cockpit  control  panel,  a  vest  (garment  containing  vibrotactile  stimulators  called  tactors  that  are 
positioned  on  the  torso),  and  a  seat  cushion  to  provide  altitude  indications.  This  garment 
provides  the  aircraft  operator  (pilot/copilot)  with  flight  control  feedback  intuitively  through  the 
sense  of  touch.  During  the  system  development  aspect  of  this  effort  (Lawson  and  Rupert,  2014), 
key  improvements  were  made  to  the  TSAS  garment,  avionic  interface,  and  software.  The  present 
report  summarizes  recent  findings  obtained  during  a  simulated  helicopter  flight  employing 
TSAS.  The  objective  was  to  evaluate  the  ability  of  TSAS  to  improve  a  pilot’s  hover  stability  in  a 
simulated  degraded  visual  environment  (DVE  or  “brownout”)  condition.  TSAS  was  integrated 
into  a  UH-60  (Black  Hawk)  full-motion  helicopter  simulator  located  at  Naval  Air  Station  (NAS) 
Patuxent  River,  MD.  Ten  pilots  evaluated  TSAS  during  flight.  Their  findings  are  reported 
below. 


TSAS  test  results 


The  test  results  were  originally  provided  to  the  Naval  Aviation  Center  for  Rotorcraft 
Advancement  (NACRA)  by  Chesapeake  Technology  International  (CTI)  at  the  conclusion  of 
SBIR  Project  Contract  No.:  N68335-09-C-0025.  The  findings  are  presented  below  with 
pennission  of  NACRA  and  CTI.  Data  Rights  to  infonnation  from  this  SBIR  are  covered  via 
standard  clauses  that  were  assigned  per  the  contract  (U.S.  Department  of  Defense,  1995). 


Objective 


To  evaluate  the  ability  of  TSAS  to  improve  a  pilot's  hover  stability  in  a  simulated  DVE. 


Methods 


TSAS  was  integrated  into  the  UH-60  simulator  at  the  Manned  Flight  Simulator  building 
located  at  NAS  Patuxent  River,  MD.  Ten  operational  pilots  each  evaluated  TSAS  during  1.5 
hour  sessions,  each  on  16  and  17  September  of  2010.  The  ADS-33E-PRF  standard  (U.S.  Army 
Aviation  and  Missile  Command,  2000)  was  used  as  the  basis  for  rating  pilot  maneuvers  and 
performance.  Navigation  data  from  the  simulation  were  forwarded  to  TSAS  over  User  Datagram 
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Protocol  (UDP)  at  50  Hertz  (Hz).  The  testing  sequence  was  done  in  three  stages  completed  by 
each  pilot: 

(1)  Pre-brief  and  familiarization:  To  determine  TSAS  capabilities  and  expected  effectiveness, 
the  ten  pilots  were  pre-briefed  on  the  system  and  test  procedures  and  then  entered  the  UH-60 
simulator  with  full  motion  activated.  They  were  allowed  time  to  familiarize  themselves  with  the 
UH-60  simulator,  the  ADS-33  course  installed  in  the  simulated  environment,  and  the  sensations 
from  TSAS.  This  familiarization  period  lasted  approximately  10  minutes. 

(2)  Test  events:  For  each  hover  test  event,  pilots  were  reset  approximately  300  feet  (ft)  from 
the  ADS-33  hover  course.  Each  pilot  then  proceeded  to  the  designated  hover  point  as  illustrated 
in  figure  1 .  Each  pilot  was  then  asked  to  maintain  position,  using  the  “hover  box”  and  traffic 
cones  as  visual  cues  to  maintain  position  and  altitude.  Once  comfortable  with  position  and 
stability  of  hover,  pilots  notified  the  evaluation  conductor  to  begin  the  test  at  which  point  a  hover 
test  scenario  started.  Afterwards,  pilots  completed  a  Cooper-Harper  Handling  Qualities  Rating 
(HQR)  form  (shown  in  figure  2)  in  the  cockpit  prior  to  starting  the  next  test  event.  Each  pilot  did 
multiple  test  scenarios  that  are  described  on  pages  5  and  6  of  this  report. 

(3)  After  the  test  scenarios  were  concluded,  each  pilot  was  debriefed  by  a  NACRA 
representative  and  comments  were  collected  during  post-flight  evaluation  as  shown  in  figure  3. 
The  evaluation  contains  ratings  of  TSAS  and  comments  on  TSAS  characteristics. 
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Figure  1.  Representation  of  hover  course. 
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Figure  2.  Handling  Qualities  Rating  form. 
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Posttlight  Questionnaire 
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Figure  3.  Post- flight  questionnaire. 

Hover  test  scenarios 

The  intent  of  the  test  was  to  compare  pilot  responses  with  and  without  TSAS  engaged,  and 
with  and  without  a  DVE.  There  were  four  hover  scenarios  planned  and  performed  by  each  pilot. 

The  four  test  scenarios  were  presented  in  the  following  order  to  each  pilot: 

(1)  Clear  visual  environment  (CVE),  no  TSAS:  Each  pilot  performed  a  60  second  (s)  baseline 
hover  test  in  CVE. 

(2)  CVE,  with  TSAS:  Each  pilot  performed  a  60  s  hover  test  in  CVE  with  TSAS  engaged  by 
evaluation  conductor. 

(3)  DVE,  no  TSAS:  Each  pilot  performed  a  60  s  hover  test  in  DVE,  with  DVE  incremented  at 
the  start  of  the  test  to  50,  70,  90,  93,  95,  97,  and  99  percent  every  10  s. 

(4)  DVE,  with  TSAS:  Each  pilot  performed  a  60  s  hover  test  in  DVE,  TSAS  engaged,  with 
DVE  incremented  at  the  start  of  the  test  to  50,  70,  90,  93,  95,  97,  and  99  percent  every  10  s.  If 
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time  was  available,  the  pilot  was  asked  to  perform  a  free  flight  around  the  airfield  for  a  few 
minutes,  and  then  asked  to  repeat  DVE  tests  for  additional  data  gathering. 

Some  pilots  performed  multiple  iterations  of  singular  test  scenarios  during  the  testing  session. 
This  was  done  to  provide  as  much  information  for  data  analysis  as  time  permitted. 

Each  test  scenario  was  standardized  so  that  each  pilot  received  the  same  DVE  conditions  during 
a  60  s  run. 


TSAS  algorithms 

TSAS  utilizes  an  onboard  configuration  file  which  is  processed  during  System  Power 
Up/Initialization  to  determine  configuration  settings.  The  configuration  file  is  in  Extensible 
Markup  Language  (XML)  format  and  is  human  readable  and  editable.  The  algorithm  variable 
settings  used  for  these  test  scenarios  were  recommended  by  Dr.  Angus  Rupert  of  the  USAARL. 
The  algorithm  is  described  in  “Configuration  Parameters  for  the  Tactile  Situation  Awareness 
System  (TSAS)”  dated  July  2010  (Configuration  Parameters,  2010). 


Findings 

The  plots  shown  below  are  summary  indications  of  the  data  collected  for  all  10  pilots.  Flight 
analysis  was  accomplished  by  determining  the  intended  hover  position  of  the  pilot  at  the 
beginning  of  the  test.  After  saving  this  position,  the  magnitude  of  deviations  in  latitude, 
longitude,  and  altitude  were  recorded  for  each  flight.  These  data  were  then  averaged  to  determine 
the  effectiveness  of  TSAS  for  all  pilots  in  the  scenarios  detailed  above. 

Pilot  hover  deviation  in  a  CVE 

Figure  4  shows  hover  deviations  that  the  pilots  exhibited  when  in  CVE  with  and  without 
TSAS.  At  the  time  of  the  maximum  deviation  from  hover,  TSAS  enabled  approximately  a  70 
percent  decrease  in  aircraft  movement.  The  chart  indicates  the  60  s  timeline  for  the  test  sequence 
for  the  X-axis  and  indicates  hover  deviation  in  feet  for  the  Y-axis. 
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Figure  4.  Average  hover  deviation  of  all  CVE  flights. 


Pilot  hover  deviation  in  a  D’VE 


Figure  5  shows  hover  deviations  that  the  pilots  exhibited  when  in  DVE  conditions  with  and 
without  TSAS.  The  brownout  conditions  were  introduced  into  the  pilot’s  field  of  view  on  a  timed 
sequence.  Observations  in  the  cockpit  indicated  that  around  97  percent,  obscuration  grew  severe 
enough  to  dramatically  affect  hover  performance.  This  was  approximately  when  the  pilot  lost 
visual  references  and  the  point  at  which  most  pilots  responded  that  they  were  uncomfortable  with 
the  hover  conditions.  At  or  beyond  97  percent  obscuration,  with  TSAS  enabled,  the  pilots  were 
able  to  maintain  a  stable  hover  position  (defined  as  a  four  foot  deviation  limit),  which  equates  to 
approximately  a  73  percent  average  decrease  in  movement  over  the  60  s  of  testing. 
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Time  of  Flight  (Seconds) 


Figure  5.  Average  hover  deviation  of  all  DVE  flights. 

Table  1  presents  cumulative  descriptive  statistical  information  for  all  tests  per  the  four  test 
scenarios  used:  CVE  (no  TSAS),  CVE  with  TSAS,  DVE  (no  TSAS)  and  DVE  with  TSAS.  Of 
note  are  three  statistics: 

a.  The  mean  distance  traveled  from  a  stable  hover  position  during  the  entire  60  s  test  scenario 
for  each  test  pilot.  Note  that  in  either  the  CVE  or  the  DVE  conditions,  the  pilots  showed  marked 
improvement  when  TSAS  was  engaged. 

b.  The  minimum  distance  traveled  from  a  stable  hover  position  during  the  60  s  scenario  (i.e., 
minimum  for  each  pilot,  averaged  across  all  pilots). 

c.  The  maximum  distance  traveled  from  a  stable  hover  position  during  the  60  s  scenario 
(averaged  across  the  pilots).  Note  the  reduction  in  deviation  for  the  DVE  condition  with  and 
without  TSAS  engaged. 

Overall,  the  descriptive  statistics  presented  in  table  1  show  that  mean  hover  deviation  in  the 
UH-60  Simulator  is  smaller  during  CVE  and  DVE  when  TSAS  is  present  (versus  when  TSAS  is 
not  assisting).  A  limitation  to  this  conclusion  is  that  it  has  not  been  supported  by  conducting 
inferential  statistics.  While  the  raw  data  were  not  available  at  the  time  of  preparation  of  this 
technical  report,  an  estimate  of  significance  was  attempted  using  the  mean  and  range  information 
in  Table  1.  We  applied  the  range  rule  to  estimate  standard  deviation  (as  the  range  divided  by 
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four),  then  carried  out  an  independent  t-test  on  the  available  means.  This  estimate  detected  a 
significantly  reduced  hover  deviation  during  CVE  flying  with  TSAS  (versus  without  TSAS,  t  = 
5.19  ,p  =  0.0001),  but  not  during  DVE  flight  (t  =  1.1  ,p  =  0.14),  probably  because  of  the  wide 
range  of  values  observed  in  the  DVE  (without  TSAS)  group.  This  should  be  considered  a  rough 
estimate,  however,  since  it  was  applied  to  a  small  sample  without  precise  information  about 
standard  deviations,  normality,  or  homogeneity  of  variance.  Also,  it  should  be  noted  that  while  a 
dependent  t-test  is  not  possible  without  the  original  data,  such  a  test  would  have  been  justified 
with  these  data  and  may  have  shown  a  statistically  significant  reduction  in  hover  deviation  under 
both  CVE  and  DVE. 

A  possible  reason  for  the  failure  to  detect  an  effect  under  DVE  is  an  insufficient  sample  of 
pilots.  An  estimate  of  statistical  power  was  conducted,  which  confirmed  that  the  study  was 
under-powered  to  detect  an  effect  at  n  =  10  (i.e.,  n  =  30  would  be  desirable  for  0.80  power).  This 
is  common  problem  for  resource-limited  operational  studies  requiring  trained  pilots.  It  would  be 
beneficial  to  gather  additional  data  under  this  flight  testing  scenario  in  future. 

Table  1. 

Pilot  AD  S3  3  hover  deviation. 


CVE 

CVEw/ 

TSAS 

DVE 

DVE  w/ 
TSAS 

Mean  (ft) 

3.1538 

0.7077 

11.2723 

4.1170 

Mean  Minimum  (ft) 

0 

0 

0 

0 

Mean  Maximum  (ft) 

5.4698 

1.6695 

79.3690 

20.2442 

Pilot  ratings 

After  each  test  event,  the  pilot  was  asked  to  complete  the  HQR  form  in  accordance  with  ADS- 
33E-PRF  standard  (U.S.  Army  Aviation  and  Missile  Command,  2000).  A  scale  ranging  from  1 
(excellent)  to  10  (major  deficiency)  was  used.  The  HQR  revealed  that  in  a  brownout  condition, 
there  was  a  trend  towards  improvement  in  pilot  rating  of  these  preliminary  test  configuration 
settings.1  Note:  The  lower  the  rating,  the  better  the  indication  that  the  pilot  could  handle  the 
aircraft.  Figure  6  and  table  2  summarize  the  HQR  data. 


1  The  same  caveats  concerning  lack  of  inferential  statistics  apply  to  these  ratings  as  above  for  hover  performance 
values;  however,  estimating  t-test  results  based  on  HQR  ratings  would  less  appropriate. 
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Average  Pilot  TSAS  Handling  Qualities  Ratings 

■  CVE  ■  CVE  w/ TSAS  ■  DVE  ■  DVE  w/ TSAS 


Figure  6.  Average  HQR. 


Table  2. 
HQR  ratings. 


CVE 

CVEw/ 

TSAS 

DVE 

DVE  w/ 
TSAS 

Mean 

3.7 

3.45 

7.90 

6.35 

Mean  Minimum 

3 

2 

5 

4 

Mean  Maximum 

5 

4 

9 

9 

For  this  study,  there  were  no  pilot  distractions  (e.g.,  emergencies,  radio  traffic,  wind,  hostile 
fire,  cockpit  distractions)  introduced  into  the  test  scenario.  In  laboratory  conditions  during  CVE, 
analysis  of  pilot  flight  characteristics  indicates  an  average  hover  deviation  of  5.46  feet  occurs 
during  60  s  of  hover  in  any  axis  without  TSAS  engaged  and  1.6  feet  with  TSAS  engaged.  While 
use  of  TSAS  results  in  a  substantial  improvement  (70  percent  to  hover  deviation),  the  actual 
distance  improved  appears  to  be  only  3.86  feet.  This  small  absolute  difference  is  to  be  expected 
under  clear  visual  flying  conditions. 

To  compare  CVE  and  DVE,  two  metrics  are  selected:  duration  and  amplitude  of  hover 
deviation.  The  metrics  are  assessed  during  the  entire  60  s  test  with  a  timed  DVE  setting  ranging 
from  0  to  99.9  percent  DVE.  These  two  metrics  address  the  following  questions:  (1)  How  long 
can  a  pilot  maintain  less  than  8  ft  (summed  in  all  axes)  of  distance  from  the  point  of  hover  entry 
into  DVE  conditions  with  and  without  TSAS  engaged?  (2)  What  is  the  average  hover  deviation 
(summed  in  all  axes)  in  DVE  conditions  with  and  without  TSAS  engaged?  Table  3  summarizes 
the  answers  to  these  questions,  and  suggests  that  TSAS  is  very  helpful  in  a  DVE.  As  brownout 
conditions  worsen,  an  increase  in  hover  deviation  is  observed  and  TSAS  reduces  a  pilot’s  hover 
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error.  Results  indicate  that  TSAS  has  the  potential  to  extend  a  pilot’s  ability  to  hover  under  DVE 
and  reduce  a  pilot’s  overall  excursion  from  a  hover  zone. 


Table  3. 

Hover  deviation  during  DVE. 


Without  TSAS 

With  TSAS 

Improvement  with  TSAS 

Duration  Hover  Deviation  is 
Maintained  within  +/-  8  ft 

10  s 

49  s 

79% 

Average  Magnitude  of  Hover 
Deviation 

11.30  ft 

4.11  ft 

63  % 

Conclusion 


This  user  evaluation  assessed  the  ability  of  TSAS  to  improve  a  pilot’s  hover  stability  in  a 
simulated  DVE.  Ten  pilots  evaluated  TSAS  during  simulated  flight  conditions.  The  flight  test 
events  consisted  of  four  hover  scenarios  that  each  pilot  performed:  CVE  with  TSAS,  CVE 
without  TSAS,  DVE  with  TSAS  and  DVE  without  TSAS.  The  magnitude  of  deviation  in 
latitude,  longitude  and  altitude  were  recorded  from  each  flight.  The  descriptive  data  showed  a 
trend  for  mean  hover  deviation  (from  the  desired  hover  point)  to  be  smaller  when  TSAS  was 
present,  although  large  variability  in  hover  performance  was  observed  in  this  small  sample  of 
pilots.  Further  pilot  testing  is  recommended. 
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AGARD  -  Advisory  Group  for  Aerospace  Research  &  Development 

CTI  -  Chesapeake  Technology  International 

CWP  -  Coalition  Warfare  Program 

CVE  -  Clear  Visual  Environment 

DoD  -  Department  of  Defense 

DVE  -  Degraded  Visual  Environment 

HQR  -  Handling  Qualities  Rating 

NACRA  -  Naval  Aviation  Center  for  Rotorcraft  Advancement 
NAS  -  Naval  Air  Station 

ORISE  -  Oak  Ridge  Institute  for  Science  and  Education 

SBIR-  Small  Business  Innovation  Research 

TP  A  -  Test  Project  Agreement 

TSAS  -  Tactile  Situation  Awareness  System 

UDP  -  User  Datagram  Protocol 

XML  -  Extensible  Markup  Language 
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